Using Function Space Theory for Understand- Ing Intermediate Layers

The representational change of input along the intermediate layers is an important aspect of understanding deep learning architectures. To this end, we propose an approach that relies on the foundation of Function Space theory. In particular, we argue that a weak-type Besov smoothness index can quantify the geometry of the clustering in the feature space of each layer. Therefore, our approach may provide an additional perspective for understanding data-models fit in the setting of deep learning. While using a different framework and perspective, the experiments we performed are in line with the results described by Tishby & Zaslavsky (2015) and Montavon et al. (2010) in the sense that for well-performing trained networks, the quality of the representation increases from layer to layer. Our approach could also be used for addressing generalization (Zhang et al., 2016), (Kawaguchi et al., 2017) as we also show that the Besov smoothness of the layer representations of the training set decreases as we add more mis-labeling. 1 FUNCTION SPACE ANALYSIS FOR NEURAL NETWORK ARCHITECTURES 1.1 FUNCTION SPACE APPROACH A function space is a class of functions bundled with a norm that assigns a non-negative magnitude to every function in the class. In many cases, we are interested in the collection of those functions for which the definition of the norm makes sense and is finite Tao (2008). For example, the functions that have a quantity nature, such as Lp spaces, or some smoothness characteristics such as Sobolev spaces. One of the practical aspects of this field, is finding the functions representation which could provide a correspondence with its Function Space. A well-known example is the representation of functions as Fourier series and the correspondence between the Fourier coefficients and the functions error decay rate. In this position paper, we will be using Geometric Wavelets for representing functions along with Besov space analysis, which is the right mathematical setup for adaptive approximation using wavelets (Dekel & Leviatan, 2005), (DeVore, 1998). As shown in (Elisha & Dekel, 2017), the weak-type Besov smoothness besov indication could describe the geometry of the clustering of the training set in the feature space of each layer. We begin with an instructive example that could demonstrate our functional perspective for neural network architectures. Assume we are presented with a set of gray-scale images of dimension √ n0× √ n0 with L class labels. Assume further that a deep network has been successfully trained to classify these images with relatively high precision. This allows us to extract the representation of each image in each of the hidden layers. To create a representation at layer 0, we concatenate the √ n0 rows of pixel values of each image, to create a vector of dimension n0. We also normalize the pixel values to the range [0, 1]. Since we advocate a function-theoretical approach, we transform the class labels into vector-values in the space RL−1 by assigning each label to a vertex of a standard simplex. Thus, the images are considered as samples of a function f0 : [0, 1]0 → RL−1. In the general case, there is no hope that there exists geometric clustering of the classes in this initial feature space and that f0 has sufficient ‘weak-type’ smoothness (as illustrated by our experiments below). Thus, a transform into a different feature space is needed. We thus associate with each k-th layer of a DL network, a function fk : [0, 1]k → RL−1 where the samples are vectors created by normalizing and concatenating the feature maps computed from each of the images. Interestingly enough, although the series of functions fk are embedded in different dimensions nk, through the